[Integration] Integrating LLM-proxy (#54/#63) onto PR stack - #73

generate_docker()/generate_workflow_docker() now recursively sweep every
.py file under the project directory into the build context, preserving
directory structure, so helper files that aren't declared as an agent
entrypoint still make it into the image. Generated dirs (docker_container/,
stubs/, grpc_stubs/) are excluded at the project root only, not at every
depth.

Stub files are placed at their agent's declared entrypoint path (mapped
from global_controller.yaml) instead of a hardcoded guess, so a stub
overwrites the exact real file it replaces. Guards against absolute and
'..'-containing entrypoints, symlinked sources, and symlinked-destination
escapes, with warnings on unsafe or unmapped stubs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…heckpoint)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

# Conflicts:
#	examples/portfolio/agents/advisor_agent.py

Users had no way to get API keys (OpenAI, Anthropic, embedding models)
into an agent container. Add a top-level `env_file` key to
global_controller.yaml pointing at a local .env file, which reaches every
container as `docker run --env-file`.

- resolve_env_file validates the path before anything launches, so a
  missing .env fails at deploy time instead of deep inside a container.
  Relative paths resolve against the project root, matching entrypoint.
- env_file_args is a context manager owning the local-vs-remote decision
  and the cleanup, so both runtimes share one code path. Local containers
  read the original file; remote containers get a copy that is deleted as
  soon as `docker run` returns, whether or not it succeeded.
- GlobalController._push_file streams the file over ssh under `umask 077`
  rather than scp, so the copy is never briefly world-readable and the
  secret never lands in a command line. _run_cmd's ssh options moved to a
  shared _ssh_args.

--env-file is appended after the explicit -e VENTIS_* flags; Docker gives
those precedence regardless of order, so a stray VENTIS_* line in
someone's .env cannot break agent wiring.

Closes #50

Two holes in the remote staging path, both found reviewing the feature
commit.

`umask 077` only governs files the shell creates, and `>` follows
symlinks -- so it did not actually guarantee a 0600 copy. The destination
path is fully predictable (`/tmp/ventis-env-ventis-ec2-<agent>-<n>`), so a
local user on the remote host could pre-create it world-readable, or point
it at a file of their own, and collect the API keys. Remove whatever sits
at the path before writing; `rm -f` unlinks a symlink rather than following
it, so `cat >` then creates a fresh file under the umask.

`_run_cmd` joins its argv with spaces and hands the result to a remote
shell unquoted. `_push_file` quoted its path but the cleanup `rm` did not,
so a container name containing a space split the `rm` into two arguments
that matched nothing -- it exited 0 while the secrets file stayed on the
host, and the returncode check logged nothing. Scrub the name down to
[A-Za-z0-9_.-] in remote_env_path, which also closes the same gap in the
`--env-file` argument and in any future use of that path.

Still open, tracked separately: a push that dies mid-transfer can leave a
copy behind, since the cleanup only covers the `docker run` that follows.
On EC2 the instance is terminated on that path, which disposes of it.

…nup-race fix (pre-pull checkpoint)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This project's agents/workflow import each other's stubs by flat module
name, not by the exporting agent's own entrypoint path. Applying
_stub_destination's entrypoint-mirroring broke both the Workflow
(ModuleNotFoundError: intent_agent) and agent-to-agent calls
(MetricsAgent -> price_agent) on live redeploy. Keeps PR #51's actual
fix (project_dir sweep for unstubbed helper files) intact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Metrics/telemetry latency scaled with instance count x per-instance
round-trip time since every instance was polled sequentially, one
blocking the next, with the following tick only starting after the
whole pass finished. Extracted the per-instance body into
_poll_one_instance (whole body wrapped in one top-level try/except,
since ThreadPoolExecutor.map() re-raises on first exception when
results are consumed) and run all instances concurrently via the same
ThreadPoolExecutor pattern _trigger_cleanup already used.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Metrics/telemetry latency scaled with instance count x per-instance
round-trip time since every instance was polled sequentially, one
blocking the next, with the following tick only starting after the
whole pass finished. Extracted the per-instance body into
_poll_one_instance (whole body wrapped in one top-level try/except,
since ThreadPoolExecutor.map() re-raises on first exception when
results are consumed) and run all instances concurrently via the same
ThreadPoolExecutor pattern _trigger_cleanup already used.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Resolved conflicts:
- metrics_agent.py: removed duplicate imports
- portfolio_workflow.py: used main's simpler intent handling
- cli.py: kept both comment explanations
- stub_generator.py: used main's version (no entrypoint injection)

Brings in all OTel cleanup work:
- Removed legacy single-destination fallback
- Removed langfuse Basic-Auth auto-injection
- Updated DESIGN.md to remove legacy references
- Simplified config env-var expansion
- Fixed langfuse config examples
- Merged latest from main (joke_writer, porting skill, etc.)

All conflicts resolved by taking feature/otel-exporter's cleaner versions.

… and remove _write_entrypoint_file

- Fixed langfuse example to use generic env-var headers pattern
- Removed _write_entrypoint_file (directory structure preservation via _sweep_py_files is cleaner)

Re-applied the parallel instance polling that was lost during conflict
resolution. The _poll_controllers method now uses ThreadPoolExecutor to
poll all instances concurrently via _poll_one_instance, preventing one
slow instance's Redis/Postgres round-trip from blocking the entire poll
tick.

…ull redeploy

Moves otel.destinations from a one-shot VENTIS_OTEL_DESTINATIONS env var
(frozen at exporter subprocess spawn) to a Redis key (otel:destinations),
mirroring the existing routing-table live-reload pattern. GlobalController
writes it at startup and again in reload_config() (SIGHUP); otel_exporter.py's
existing 5s poll tick re-reads it each cycle and rebuilds its
BatchSpanProcessors only when it changed. No signal-forwarding, no subprocess
restart, no ProcessSupervisor.restart -- just a small ProcessSupervisor.is_registered()
so reload_config knows whether the exporter is even running.

Kept in scope: exporter start-gating at boot is unchanged (still skipped
entirely if otel.destinations is absent at startup); destinations added after
boot only take effect if the exporter was already running.

Drops the redis-connection-info env plumbing (VENTIS_REDIS_HOST/PORT/DB,
GlobalController._otel_exporter_env) added in the previous commit --
otel_exporter and GlobalController always run on the same host, and
RedisClient's own defaults already are localhost:6379/db0, so passing
them through was dead flexibility for a case that doesn't exist yet.

Kept comments only where behavior is genuinely non-obvious (why the
exporter polls Redis instead of restarting, why reload_config gates on
is_registered, the invalid-update-keeps-old-processors fallback). Dropped
comments/docstrings that just narrated 'this was added' on trivial
pass-through code.

Everything under ventis/ and tests/ that the canyonos CLI branch depends
on, lifted off feature/config-reloading with no cli/ or examples/ changes.

- Package layout: move deploy/future/ventis_context/bedrock/utils under
  ventis/controller/, add ventis/Dockerfile and ventis/README.md.
- ventis/server.py: Flask control surface the CLI's container talks to
  (/deploy, /clean, /status), replacing the ad-hoc entrypoint.
- ventis/cli.py: fold `build` into `deploy`, support the .car artifact
  layout (.car/app sources, .car/config declarations, .car/stubs), and
  resolve env_file against the project dir so it matches how the
  GlobalController resolves it at runtime.
- GlobalController: persist a dashed-uuid project_id and publish the
  controller identity to Redis.
- OTLP exporter: generate Future.id at 64 bits (secrets.token_hex(8)) so
  it is a valid OTel span_id without truncation, and cost lookups that
  fail (no pricing table on a local deploy) now cost at 0 instead of
  dropping the whole telemetry row.
- stub_generator: a stub is written to exactly one location, the path of
  the entrypoint it replaces, rather than being duplicated at the flat
  basename as well. Flat is only the fallback for a stub with no
  entrypoint mapping or one whose mapping escapes the build context.
  Carries over the placement half of 692d17c from feature/all-the-files,
  which never reached this line; the entrypoint-adjacent YAML discovery
  from that commit is deliberately left out, since the .car layout
  already resolves declarations from .car/config.
- stub_generator: fail loudly when an agent has no declaration or
  entrypoint.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Stub placement is now mirrored-only, so a workflow importing the flat
basename no longer resolves -- the stub is written to agents/<name>.py
and nothing is left at the context root. Switch the four example
workflows to the nested form.

Cherry-picked from 7f925ef on fix/remove-duplicate-stub.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Cherry-picked from a0efb5a on bug/local-provider-routing, resolved
against the .car/CLI packaging changes already on this branch.

Agents used to be reached at host.docker.internal:<host_port>, which
only works when whatever is doing the reaching shares the host's network
namespace. They now join a `ventis-local` docker network and are
addressed by container name at the fixed container port, so routing no
longer depends on the caller's vantage point.

- Local/_runtime.py: --network ventis-local instead of --add-host,
  VENTIS_AGENT_PORT is the container port, VENTIS_AGENT_HOST is the
  container name, routing_endpoint_for returns <runtime_id>:50051.
- global_controller.py: creates the network alongside the local Redis
  container, and derives status/metrics Redis keys from
  instance_manager._routing_endpoint_for instead of the host string.

Conflict resolution notes:
- Kept this branch's MAX_PORT_ATTEMPTS retry loop and env_file support
  around the docker run, applying the new network/env flags inside it.
- Kept _is_local_host in Local/_runtime.py; a0efb5a dropped it, but
  env_file_args (added later on the CLI line) still needs it.
- The controller:<endpoint>:agent_id key written after launch, which
  postdates a0efb5a, now uses the container-name endpoint so it matches
  the keys global_controller reads back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The CLI layer on top of the ventis fixes: `canyonos` wraps the Global
Controller in a container and drives it over HTTP, so a project goes
from source to a running workflow with a local dashboard in one command.

- cli/: the canyonos package -- deploy (which folds in init, sync, build
  and launch, then auto-starts the dashboard once the workflow reports
  up), serve, stop, logs, quit, clean, config, integrate, new-app.
- cli/canyonos/dashboard.compose.yml: api/db/web stack. The api port is
  published so the GC container can POST OTLP spans to /v1/traces, which
  is also what renames ventis' `project_id` attribute to the
  `canyon.project.id` the dashboard queries filter on. serve replaces the
  api container every run, since it reads the controller's Redis identity
  only at startup and would otherwise keep serving a stale project.
- cli/canyonos/constants.py: resolve the config path per call rather than
  at import, preferring .car/config over the flat layout.
- .claude/skills/porting-to-canyonos-core/: the porting skill `canyonos
  integrate` installs, plus its validator.
- examples/: joke_writer converted to the .car layout, epigenomics added,
  and workflow imports pointed at each agent's entrypoint path to match
  single-location stub placement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Re-apply the feature/llm-proxy-telemetry proxy onto the current CLI
packaging architecture (rather than merging the stale branch):

- Add ventis/llm_proxy/ package (imports remapped to current layout,
  with in-container flat fallbacks for redis_client/ventis_context)
- Delete ventis/controller/bedrock.py; rewrite the 3 examples to call
  boto3 bedrock-runtime converse() directly (telemetry now via proxy)
- Start the proxy per-container in LocalController and auto-inject the
  X-Ventis-Future-ID boto3 header
- Inject AWS_ENDPOINT_URL_BEDROCK_RUNTIME in Local/EC2 runtimes
- Copy llm_proxy into agent+workflow images via stub_generator; add
  flask/requests to agent image + host deps
- Update FUTURE_SCHEMA.md provenance and instance-manager runtime tests

…uest

before-call handlers receive the prepared-request params dict, not the
request object (that only exists on before-send). Reading kwargs['request']
was always None, so X-Ventis-Future-ID was never attached and the proxy
could not attribute token telemetry to the executing future.

Bring the CLI architecture doc (build/deploy/config flows and container
lifecycle) over from cli/canyonos-cli, and drop the stale PyPi republish
snippet from the README.